Remarks 

The above Amendments and these Remarks are in reply to the OflSce Action mailed 
August 29, 2001. 

Currently, claims 11-142 are pending. 

I. Rejections of Existing Claims 

Claims 11-99 have been rejected under 35 U.S.C. § 103(a), as being obvious in view 
of Moezzi et al (U.S. Patent No. 5,850,352). Because Moezzi et al does not disclose, teach 
or suggest all of the limitations of Applicants' claims. Applicants respectfully assert that the 
currently pending claims are in condition for allowance. 

A telestrator system " is a system that allows an operator to draw on live or stored 
video." (Specification at page 2). Various embodiments of Apphcants' invention pertains to 
an: 

improved telestrator system that allows a broadcaster or other 
operator to aimotate video during or after an event. For 
example, while televising a sporting event (or other type of 
event), an announcer (or other user) can use the present 
invention to draw over the video of the event to highlight one 
or more actions, features, etc. In one embodiment, when the 
xiser draws over the video it appears that the user is drawing on 
the actual field or location of the event. (Specification at page 
4)- 

For example: 

In operation, a user of the system can use a finger on touch 
screen 104 to draw a graphic (e.g. shape or curve). Information 
about what is being drawn is sent to telestration processor 108. 
The resulting blend of the graphic with the video is sent back to 
display 102. (Specification at page 8). 

In another example, a pre-defined shape can be added to the video instead of a hand 
drawing: 

Examples of highlights could include a cloud, a hollow 
circle, icons, team logos, bright spots, etc. The user interface 
could allow the operator to choose the appropriate highlight. In 
operation, the one touch highUght can be performed by the 
operator touching the screen once quickly. Upon sensing the 
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quick ^Pbh, the system will add the pre-chosen^phlight to the 

video at the touched position. (Specification at page 34). 



In addition to a telestrator system, Applicants' invention can pertain to other systetns 
for adding images to video. 

One embodiment of the present invention allows a user of the system to add an image 
to the video such that the image appears to be drawn directly on a stirface of the scene being 
depicted in the video: 

For example, during an American football game, if a 
user draws on the video it q)pears that the user is actually 
drawing on the playing field. After the user draws the 
illustration, the drawing will appear to be painted on the 
playing field so that if the camera moves or the broadcaster 
changes cameras the drawing still appears on the playing field 
in the appropriate perspective. This feature is depicted in 
Figures 3 and 4. Figure 3 shows a video firame 1 6 which is 
broadcast as a result ofusing the present invention. The 
operator had drawn an arrow 12 to show the path of a player or 
object on the playing field. Figure 4 shows a video fiWe 18 
which is also the result of using the present invention. 
However, Figure 4 shows arrow 12 at a different position in 
firame 1 8 than in fi:^e 1 6. That is because after the user had 
drawn arrow 12 and it was depicted in frame 16, the camera 
panned down the playing field. Instead of being pointed 
between the 20 and 1 5 yard lines, the camera is now pointed 
directly at the 20 yard line and, thus, the 25 yard line is now 
visible. Because the field moved within the camera's field of 
view, the location of arrow 12 also moved within the camera's 
field of view. Arrow 12 now appears to be drawn directly on 
^ the playing field as opposed to on the television monitor. As 
the field moves on the video screen, arrow 12 will also move. 
(Specification at page 7). 

Claim 1 1 recites various aspects of the present invention. For example, claim 1 1 

recites: 

11. A method for annotating video, comprising the 
steps of: 

receiving video, said video depicts a surface at a Uve 

event; 

receiving a graphic manually created by a human 
operator during said live event; and 

blending said graphic with said video such that said 
graphic appears to be drawn on said surface, said step of 
blending being performed during said Uve event. 
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The cited prio^Pr does not disclose receiving a graphic^Pi is **manually created by 
human operator during said live event..." Additionally, the cited prior art does not blend 
graphic "such that said grq)hic appears to be drawn on said surface" as described in th 
above quotation fix>m the specification. 

Moezzi et aL discloses a system that provides for: 

telepresence and immersive video, being the non-real-time 
creation of a synthesized, virtual, camera/video image of a real- 
world scene, typically in accordance with one or more viewing 
criteria that are chosen by a viewer of the scene. The creation 
of the virtual image is based on a computerized video 
processing-in a process called hypOTnosaicing— of midtiple 
video views of the scene, each firom a different spatial 
perspective on the scene. {Moezzi et aL, col. 9, lines 10-18). 

In one of its aspects, the present invention is embodied 
in a method of telepresence, being a video representation of 
being at real-world scene that is other than the instant scene of 
the viewer. The method includes (i) capturing video of a real- 
world scene firom each of a multiplicity of different spatial 
perspectives on the scene, (ii) creating firom the captured video 
a fiiU three-dimensional model of the scene, and (iii) producing, 
or synthesizing, firom the three-dimensional model a video 
representation on the scene that is in accordance with the 
desired perspective on the scene of a viewer of the scene. 
{Moezzi et aL, col. 9, lines 47-57) 

This method is thus called "immersive telepresence" 
because the viewer can view the scene as if immersed therein, 
and as if present at the scene, all in accordance with his or her 
. desires. Namely, it appears to the viewer that, since the scene is 
presented as the viewer desires, the viewer is immersed in the 
scene. Notably, the viewer-desired perspective on the scene, 
and the video representation synthesized in accordance with 
this viewer-desired perspective, need not be in accordance with 
any of the video captured fi-om any scene perspective. {Moezzi 
et aLy col. 9, lines 58-67). 

To help imderstand the technology, Moezzi et aL provides an example of its use: 

In the example of an American football game, the viewer/user 
could view the game in progress as if he or she was a player 
"inside" the game, even to tilie extent of looking "outward" at 
the stadium spectators. {Moezzi et aL, col. 16, lines 20-25) . . . 

A viewer interface is responsive to a viewer of the scene to 
receive a viewer selection of a desired view on the scene. This 
selected and desired view need not be identical to any views 
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that aiHRthin any of the multiple received vid^images ... a 

visiializer generates (alternatively, "synthesizes") (alternatively 
"extracts") fipom the three-dimensional model of the scene, and 
in accordance with the received desired view, a video image on 
the scene that so shows the scene from the desired view. 
(Moezzi et al, col. 20, lines 30-39) 

As can be seen, Moezzi et al pertains to a system that can create piece together video 
from different cameras of a real world scene to create a new view of that real world scene 
based on the video from different cameras. The invention of Moezzi et al is not focused on 
annotating video, adding a new gr^hic drawn by a human to the video or adding a graphic so 
that it appears to be drawn on the surface. As such, Moezzi et al does not disclose or teach 
all of the limitations of claim 1 1 . 

In the office action, the Examiner states that Figures 9a-9c of Moezzi et al disclose 
the step of **receiving a graphic manually created by a human operator." However, a review 
of those figures in Moezzi et al reveal otherwise. That is. Figures 9a, 9b and 9c depict a 
gnq)hical rendition of the 3D environment model. These figures do not show drawings 
created by a human operator. Figure 9a does show a person as a low resolution cyhnder like 
object: 

People in the scene are detected and modeled as cylinders in 
oiu: current implementation as shown in Fig. 9a. {Moezzi et al, 
col. 45, lines 26-29). 

The cjdinder referred to in the above quotes and depicted in Fig. 9a is not disclosed to be a 
gr^hic manually created by a human operator. Rather, it appears to be a computer rendered 
low resolution image of a person at the scene. A similar analysis appUes to Figs. 9b and 9c: 



FIG. 9b is a graphical rendition of the full 3D 
environment model generated by the environmental model 
builder of the immersive video system of the present invention 
for an indoor karate demonstration as was previously shown in 
FIG. 4, the two human participants being clothed in karate 
clothing with a kick in progress, the scale and the resolution of 
the model being clearly observable. 

FIG. 9c is another graphical rendition of the full 3D 
environment model generated by the enviromnental model 
bxiilder of the immersive video system of the present invention, 
this time for an outdoor karate demonstration, this time the 
environmental model being further shown to be located in the 
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static i^e, particularly of an outdoor courtyar^Pl/oezz/ et al , 

col. 21, lines 20-34). 



Therefore, Figs 9a-9c do not show a graphic drawn by a human operator. 

The Examiner also states that Figs 9a-9c oiMoezzi et al disclose blending the grqjhic 
with the video. First, as discussed above, of Moezzi et al does not disclose blending a 
graphic drawn by a human. Second, Figs. 9a-9c of Moezzi et al do not disclose blending a 
graphic at all. The fuzzy image in those figures is an actual person at the scene, not a new 
image to be added to the video. Furthermore, Figs 9a-9c represent models of the 
environment, they are not the final video. 

Applicants also note that the Examiner does not provide evidence that all of the 
limitation of claim 1 1 are obvious in light of Moezzi et al That is, the Examiner quotes the 
last step as **blending said graphic with said video." However, the last step of claim 11 also 
recites the language: "such that said graphic appears to be drawn on said surface." This 
additional language is not addressed by the Examiner. Applicants assert that Moezzi et al 
does not disclose the blending of a video "such that said graphic appears to be draw on said 
surface." Rather, as explained above, Moezzi et al discloses the creation of a virtual image 
based on real images. One example in Applicant's specification of blending such that the 
graphic appears to drawn on the surface is the airow depicted as painted on the field in Figs 3 
and 4. As the camera pans, the arrow moves with the field giving the illusion that the arrow 
is painted on the field. This concept is not taught or suggested by Moezzi et al 

For all of the reasons discussed above. Applicants respectfully assert that claim 11 
and all claims that depend fi-om claim 1 1 are patentable over the cited prior art. 

Claun 33 recites the step of '^receiving a graphic manually created by a human 
operator..." Thxis, for the same reasons as discussed above with respect to claim 11, 
Applicants assert that claim 33 and all claims that depend from claim 33 are patentable over 
the cited prior art. Claim 33 also recites that at least a portion of the surface captured in the 
video is occluded and that the graphic is blended without drawing over the object that is 
occluding the surface. This concept is not taught or suggested by Moezzi et al nor has this 
limitation been addressed by the Examiner, therefore, providing another basis for 
patentability. For all of the reasons discussed above. Applicants assert that claim 33, and all 
claims that depend from claim 33, are patentable over the cited prior art. 

Claim 42 recites the steps of receiving two dimensional position information for a 

graphic, converting two dimensional positions to three dimensional positions and converting 
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the three dimensiona^Psitions to two dimensional positions, process is not disclosed or 
suggested by Moezzi et al Applicants assert that the Examiner has not made a prima facie 
case of obviousness for claim 42 because the Examiner did not address the above mentioned 
limitations regarding converting. Therefore, Applicants assert that claim 42, and all claims 
that depend fix>m claim 42, are patentable over the cited art. 

Claim 53 recites the receiving of a drawing manxially created by a human. Thus, 
claim 53 is patentable for the respective reasons discussed above with respect to claim IL 
Claim 53 also recites the step of "smoothing said drawing." This step of smoothing is not 
taught or suggested by Moezzi et al. AppUcants assert that the Examiner has not made a 
prima facie case of obviousness for claim 53 becaxise the Examiner did not address the above 
mentioned limitations regarding smoothing. Therefore, AppUcants assert that claim 53, and 
all claims that depend horn claim 53, are patentable over the cited art. 

Claim 56 recites a series of steps that are used to blend images using blending values. 
There is no disclosure in Moezzi et aL to blend images using such blending values. 
AppUcants assert that the Examiner has not made a prima facie case of obviousness for claim 
56 because the Examiner did not address the above mentioned limitations regarding blending 
values. Therefore, AppUcants assert that claim 56, and all claims that depend from claim 56, 
are patentable over the cited art. 

Claim 69, and all claims that depend from claim 69, are patentable over Moezzi et al 
for similar reasons to those discussed above with respect to claim 56, 

Claim 79, and all claims that depend from claim 79, are patentable over Moezzi et al 
for similar reasons to those discussed above with respect to claim 11. 

Claim 85, and all claims that depend from claim 85, are patentable over Moezzi et al 
for similar reasons to those discussed above with respect to claim 33. 

Claim 92, and all claims that depend from claim 92, are patentable over Moezzi et al 
for similar reasons to those discussed above with respect to claim 98. 

Claim 98, and all claims that depend from claim 98, are patentable over Moezzi et al 
for similar reasons to those discussed above with respect to claim 53. 

n. New Claims 

Applicants have added new claims 100-142. Claims 100, 115 and 124 are 
independent claims. Claim 100 recites the steps of "manually adding a graphic to said video 
during said Uve event; and maintaining said graphic in said video such that said graphic 
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appears to be drawn (M&id surface/' As discussed above with^pect to claim 1 1, Moezzi et 
al does not disclose or suggest the adding a graphic to a video such that the graphic appears 
to be drawn on said surface. Claims 101 -142 are patentable for similar reasons. 

m. Information Disclosure Statements 

On October 21, 1999, when fiUng the present application. Applicant filed an IDS. A 
copy of that IDS is attached as Exhibit A to this Response, Attached as Exhibit B to this 
Response is a copy of a postcard received from the United States Patent and Trademark 
Office CTTO") acknowledging receipt of the IDS and the cited prior art references. 

Apphcants have not received the form PTO-1449 fiom the Examiner, acknowledging 
that the Examiner considered the cited prior art. Apphcants assert that the Examiner has a 
duty to consider the cited prior art and return a copy of the form PTO-1449 to the Applicants. 
Therefore, Apphcants respectfully request that the Examiner send to Apphcants a copy of the 
initialed PTO-1449 acknowledging that the Examiner considered the cited prior art. 

In paragraph 1 of the Office Action, the Examiner acknowledged that a second IDS 
submitted on February 12, 2001 is missing fi-om the file. Attached to this Response as 
Exhibit C is a copy of the IDS submitted on February 12, 2001 . Attached to this Response as 
Exhibit D is a copy of the postcard received fi^om the USPTO acknowledging receipt of the 
IDS and the references cited in the IDS. Apphcants respectfully request that the Examiner 
send to Apphcants a copy of the initialed PTO-1449 acknowledging that the Examiner 
considered the cited prior art. 

IV. Amendments Do Not EfTect Patentability 

Apphcants added new claims 100-142 to more sufficiently cover additional 
embodiments. The addition of these new claims is not related to the patentability of claims 
11-99. 

Apphcants made minor amendments to claims 11, 15, 33, 35, 43, 56, 62, 65, 66, 69, 
72, 76, 79, 85, 87, 92, and 98. These amendments are not related to patentabiUty, and are not 
presented in response to a rejection. Rather, they correct typographical errors. For example, 
the amendments to claims 1 1 and 33 changed "said" to -a—, which is not in response to any 
rejection fix>m the Examiner. The amendment to claim 15 changed ""a" to -said-, which is 
not in response to any rejection from the Examiner. The amendments to claims 35 and 87 
change '^blending values" to -a first blending value-, which is not in response to any rejection. 
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The amendments to IBLs 43, 62, 66, 66, 72 and 76 fix a and do not address any 
rejections. The amendments to claims 79, 85, 92, and 98 changed '^processors progranmied 
to perform" to -one or more processors perform-, which is not in response to any rejection 
fix>m the Examiner and broadens the claims. Claims 56 and 69 were sUghtly reworded to 
cover embodiments determined to be valuable to Applicants. The amendments to claims 56 
and 69 are is not in response to any rejection &om the Examiner. The remarks submitted to 
demonstrate the claims 56 and 69 are patentable apply to both the original claims and the 
amended claims because both the original claims and the amended claims are patentable over 
the cited prior art. 

CONCLUSION 

In view of the above Amendments and Remarks, reconsideration of claims 11-99 is 
requested. Consideration of newly added claims 100-142 is also requested. 

The Conmiissioner is authorized to charge any underpayment or credit any 
overpayment to Deposit Account No. 501826 for any matter in cormection with this 
Response, including any fee for extension of time, which may be requested. 

Respectfully submitted, 

~Burt Magen ^ 
Reg. No. 37,175 



Date: 



ViERRA Magen Marcus Harmon & DeNiro LLP 
685 Market Street, Suite 540 
San Francisco, California 94105 
Telephone: 415.369.9660 
Facsimile: 415.369.9665 



Attomey Docket No.: SPTV-OI038USO 
sptv/l038/1038jesp^l 



-36- 



APPENDIX 



1 1 . (once amended) A method for annotating video, comprising the steps of: 
receiving video, said video depicts a surface at [said] a hve event; 
receiving a grq)hic manually created by a human operator during said hve event; and 
blending said gr^hic with said video such that said gr^hic appears to be drawn on 
said surface, said stq> of blending being performed during said hve event 

1 5 . (once amended) A method according to claim 1 1 , wherein: 

prior to said step of blending, at least a portion of said video depicts at least a portion 

of said siuiace at [a] said hve event and a set of one or more objects occluding said surface; 

and 

said step of blending includes blending said graphic with said portion of said video 
without drawing said graphic over said objects, 

33. (once amended) A method for annotating video, comprising the steps of: 
receiving video, at least a portion of said video depicts at least a portion of a surface at 

a live event and a set of one or more objects occluding said siuface; 

receiving a graphic manually created by a human operator during said hve event; and 
blending [a] said gr^hic with said portion of said video without drawing said graphic 

over said objects, said steps of receiving video, receiving a graphic and blending are 

performed during said hve event. 

35. (once amended) A method according to claim 34, further comprising the steps 

of: 

receiving a selection of a portion of an image, said portion of said image includes a 
set of colors; 

receiving [said] a first blending value[s] for said set of colors; and 
storing said first blending values. 
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43. (onc^Kiended) A method according to clHP42, further comprising the 
steps of: 

receiving camera sensor data for a first camera, said video being captured by said first 
camera, said step of converting said one or more three dimensional locations to a second set 
of one or more two dimensional positions is performed [uses] using said camera sensor data 
for said first camera. 



56. (once amended) A method for blending images, comprising the steps of: 
storing blending values for a set of colors; 
receiving a first image [after said step of storing]; 
receiving a second image after said step of storing; 
accessing color data for said second image; 

comparing said color data to said [blending values for said] set of colors; and 
causing a blending of said first image with said second image based on using [said] 

one or more blending values for [said set of colors.] colors of said set of colore that match 

said color data for said second image. 

62. (once amended) A method according to claim 56, wherein: 
said steps of accessing and comparing include the steps of: 

accessing a first pixel in said second image, said first pixel having a first color 

value, 

accessing additional pixels nearby to said first pixel in said second imaige, said 
additional pixels having additional color values, 

comparing a first color value for said first pixeffo a color map, said color map 
stores color values for said set of colors and said blending values for said set of colors, 

comparing said additional color values to said color m^, 

identifying a first blending value corresponding to said first color value based 
on said color map, 

identifying additional blending values corresponding to said additional color 
values based on said color map, 

calculating an average of said first blending value and said additional blending 

values, and 

adjusting said average based on a weighting factor, and 
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said step of ^^ing a blending includes using said adj^^bd average to blend said fiist 
pixel with a corresponding pixel in said first image. 



65. (once amended) A method according to claim 56, wherein: 
said set of colors includes multiple visibly distinct colors. 

66. (once amended) A method according to claim 56, wherein: 

said step of accessing color data includes accessing color data for a first portion of 
said second image, said first portion of said second image corresponds in position to said first 
image; and 

said step of causing a blending includes causing said first image to be blended with 
said first portion of said second image and not oih&r portions of said second image. 

69. (once amended) An apparatus, comprising: 
one or more processors; 

an input device in commimication with said one or more processors; 
an output device in commxmication with said one or more processors; and 
at least one storage device in communication with said one or more processors, said 
one or more processors [programmed to preform] perform a method comprising the steps of: 

storing blending values for a set of colors, 

receiving a first image, 

receiving a second image, 

accessing color data for said second image, 

comparing said color data to [said blending values for] said set of colors, and 
causing a blending of said first image with said second image based on using 

[said] one or more blending values for [said set of colors.] colors of said set of colors that 

match said color data for said second image. 

72. (once amended) An apparatus according to claim 69, wherein: 
said steps of accessing and comparing include the steps of: 

accessing a first pixel in said second image, said first pixel having a first color 

value. 
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accessl^additional pixels nearby to said first p^t in said second image, said 
additional pixels having additional color values, 

comparing a first color value for said first pixel to a color map, said color map 
stores color values for said set of colors and said blending values for said set of colors, 

comparing said additional color values to said color map, 

identifying a first blending value corresponding to said first color value based 
on said color map, 

identifying additional blending values corresponding to said additional color 
values based on said color map, 

calculating an average of said first blending value and said additional blending 

values, and 

adjusting said average based on a weighting factor; and 
said step of causing a blending includes xising said adjusted average to blend said first 
pixel with a corresponding pixel in said first image. 

76. (once amended) An apparatus according to claim 69, wherein: 

said step of accessing color data includes accessing color data for a first portion of 

said second image, said first portion of said second image corresponds in position to said first 

image; and 

said step of causing a blending includes causing said first image to be blended with 
said first portion of said second image and not other portions of said second image. 

79. (once amended) An apparatus, comprising: 
one or more processors; 

a drawing device in communication with said one or more processors; 
an output device in communication with said one or more processors; and 
at least one storage device in commxmication with said one or more processors, said 
one or more processors [programmed to preform] perform a method comprising the steps of: 
receiving video, said video depicts a surface at [said] a live event, 
receiving a graphic manually created by a human operator during said live 

event, and 
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causinplF blading of said graphic with said^^eo such that said graphic 
appears to be drawn on said surface, said step of blending being performed during said live 
event. 

85. (once amended) An apparatus, comprising: 
one or more processors; 

a drawing device in communication with said one or more processors; 
an output device in conmiimication with said one or more processors; and 
at least one storage device in conununication with said one or more processors, said 
one or more processors [programmed to preform] perform a method comprising the steps of: 

receiving video, at least a portion of said video depicts at least a portion of a 
surface at a Uve event and a set of one or more objects occluding said surface, 

receiving a graphic manually created by a human operator during said live 

event, and 

causing a blending of a gr^hic with said portion of said video without 
drawing said graphic over said objects, said steps of receiving video, receiving a gr^hic and 
causing are performed during said live event. 

87. (once amended) An apparatus according to claim 86, wherein said method 
further includes the steps of: 

receiving a selection of a portion of an image, said portion of said image includes a 
set of colors;. 

receiving [said] a first blending value[s] for said set of colors; and 
storing said first blending value[s] . 

92. (once amended) An apparatus, comprising: 
one or more processors; 

a drawing device in conmiunication with said one or more processors; 
an output device in conmiunication with said one or more processors; and 
at least one storage device in commimication with said one or more processors, said 
one or m ore processors [programmed to preform] perform a method comprising the stq)S of: 
receiving video, said video depicts a surface at said live event. 
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receivi^t^] two dimensional position infomie^|^ for at least a portion of a 
graphic created in relation to a two dimensional image, 

converting a first set of one or more two dimensional positions to one or more 
three dimensional locations, said first set of one or more two dimensional positions 
correspond to said two dimensional position information, 

converting said one or more three dimensional locations to a second set of one 
or more two dimensional positions, and 

causing a blending of said gr^hic with said video based on said second set of 
one or more two dimensional positions. . 

98. (onceamraded) An apparatus, comprising: 
one or more processors; 

a drawing device in communication with said one or more processors; 
an ou^ut device in communication with said one or more processors; and 
at least one storage device in communication with said one or more processors, said 
one or more processors [progranmied to preform] perform a method comprising the steps of: 
receiving video, said video dq)icts a surface at said live event, 
receiving at least a portion of a drawing manually created by a hiunan operator 
during said Uve event, 

smoothing said drawing, and 

blending said smoothed drawing with said video dxuing said live event. 
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